Skip to content

[SPARK-52646][PS] Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode #51370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[SPARK-52646][PS] Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode #51370

wants to merge 8 commits into from
+61 −2

Conversation

xinrong-meng
Copy link
Member

@xinrong-meng xinrong-meng commented Jul 4, 2025 •
edited
Loading

What changes were proposed in this pull request?

Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode

Two main changes:

  • Numeric vs. string: Previously, 1 == "1" returned True; now it returns False, matching pandas behavior. Also "1" == 1 returns False.
  • Non-bool numeric vs. bool: Previously, 1 == True raised an error; now it evaluates to True, so does True == 1.

Why are the changes needed?

Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52556.

Does this PR introduce any user-facing change?

>>> import pandas as pd
>>> import numpy as np
>>> 
>>> ps.set_option("compute.fail_on_ansi_mode", False)
>>> ps.set_option("compute.ansi_mode_support", True)
>>> psidx1 = ps.Index([1, 2, 3])
>>> 
>>> psidx2 = ps.Index([1.0, 2.0, 3.0])
>>> psidx1 == psidx2
Index([True, True, True], dtype='bool')
>>> psidx2 == psidx1
Index([True, True, True], dtype='bool')
>>> 
>>> psidx3 = ps.Index(["1", "2", "3"])
>>> psidx1 == psidx3
Index([False, False, False], dtype='bool')
>>> psidx3 == psidx1
Index([False, False, False], dtype='bool')
>>> 
>>> psidx4 = ps.Index([True, False, True])
>>> psidx1 == psidx4
Index([True, False, False], dtype='bool')
>>> psidx4 == psidx1
Index([True, False, False], dtype='bool')

How was this patch tested?

Unit tests.

(dev3.11) spark (comp_op) % SPARK_ANSI_SQL_MODE=false  ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_comparison_dtype_compatibility"
...
Tests passed in 5 seconds
(dev3.11) spark (comp_op) % SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_comparison_dtype_compatibility"
...
Tests passed in 5 seconds

Was this patch authored or co-authored using generative AI tooling?

No.

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-52646][PS] Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode [SPARK-52646][PS] Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode Jul 22, 2025
@xinrong-meng xinrong-meng marked this pull request as ready for review July 22, 2025 00:24
@xinrong-meng
Copy link
Member Author

@ueshin may I get a review please?

xinrong-meng
xinrong-meng commented Jul 22, 2025
View reviewed changes
python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
self.assert_eq(pdf["int"] == pdf["bool"], psdf["int"] == psdf["bool"])
self.assert_eq(pdf["bool"] == pdf["int"], psdf["bool"] == psdf["int"])
self.assert_eq(pdf["int"] == pdf["float"], psdf["int"] == psdf["float"])
if is_ansi_mode_test: # TODO: match non-ansi behavior with pandas
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://issues.apache.org/jira/browse/SPARK-52644

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
PANDAS API ON SPARK PYTHON
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@xinrong-meng